So, in this pr, we had removed all static random generator and only share a static seed between different modules(because we want to make all the module have a fixed seed for the unit test). But, however, there are many tests(unit and meta test) failed for these changes. On locally, I have:

The following tests FAILED:
1:26 PM 212 - unit-NeuralLinearLayer (Failed)
1:26 PM 246 - unit-QuadraticTimeMMD (Failed)
1:26 PM 247 - unit-TwoDistributionTest (Failed)
1:26 PM 278 - integration_meta_cpp-clustering-gmm (Failed)
1:26 PM 282 - integration_meta_cpp-distance-cosine (Failed)
1:26 PM 299 - integration_meta_cpp-multiclass_classifier-random_forest (Failed)
1:26 PM 303 - integration_meta_cpp-neural_nets-feedforward_net_classification (Failed)
1:26 PM 304 - integration_meta_cpp-neural_nets-feedforward_net_regression (Failed)

All of them are rely on global random which has been removed from this and last pr. We wondering if they failed because the side effect of global random removal and we could assume(believe) we haven't hurt those modules inside after apply this pr.
Hi @karlnapf and @lambday , could you take a look at this pr? About global random and these the way to set and use random generator, please tell me if there has anything you don't like, thanks a lot!

@MikeLing i've rebased the feature/random-refactor over the current develop so you have the fix from @karlnapf for the global functions... you would need to clear up a bit branch to be able to merge/test :)

Hi @karlnapf , so we still have these meta tests failed:

296 - integration_meta_cpp-neural_nets-feedforward_net_regression (Failed)
297 - integration_meta_cpp-neural_nets-feedforward_net_classification (Failed)
320 - integration_meta_cpp-multiclass_classifier-random_forest (Failed)
324 - integration_meta_cpp-clustering-gmm (Failed)

Here is the generated data of gmm : https://pastebin.mozilla.org/9026925
and here is the reference data of it: https://pastebin.mozilla.org/9026926

Do we want to ensure all the difference is the side effect of global random and we don't need to worry about?

@MikeLing as for the

296 - integration_meta_cpp-neural_nets-feedforward_net_regression (Failed)
297 - integration_meta_cpp-neural_nets-feedforward_net_classification (Failed)
320 - integration_meta_cpp-multiclass_classifier-random_forest (Failed)
324 - integration_meta_cpp-clustering-gmm (Failed)

I think all of those contain random elements, so the integration tests indeed need to be updated if you change the seed.

Hi @vigsterkr and @karlnapf , I changed these tests' expected output depend on linux(both gcc and clagn), but the window's random implementation looks like different :(

The most annoying solution I can think of is using #ifdef to account for both systems ...

ok so i mean we are not thaaaaaaaaaaat bad (although of course note that good either). so basically i've just tested and as it is specified in the standards the random engines (mt19937, mt19937_64, minstd_rand etc) are generating the same values with the same order on the same seed :)

of course it's annoying now that we'll have to provide our distribution mappers (like uniform, normal etc) but still we can do that - would be nice though to use the std api - and then have that as our random. still we can this way through out the external random generators that i've pulled in couple of years ago and still we are good with removing the global random :)

so basically i've just tested and as it is specified in the standards the random engines (mt19937, mt19937_64, minstd_rand etc) are generating the same values with the same order on the same seed :)

I just test it on Linux(based on gcc, run on CLion) and window(on visual studio), and the results displayed totally different https://gist.github.com/MikeLing/12cf9254248e9ce49d343f297e920e1e#gistcomment-2154781

@MikeLing :) cool console!
but what i have tested was this:

    std::mt19937_64 rng(100);
    for(int32_t i=0 ; i<15; i++)
    {
        std::cout << rng() << std::endl;
}

based on the standard this should be the same regardless of the compiler/os/etc.
these random engines are all generating uniform distributed integers. now the way you generate a given interval of uniform distributed elements or other distributions based on these engines is actually implementation specific. but that we can provide ourselves :) see in CRandom how we use a PRNG that generates uniform distributed random to get samples of a normal distribution with a given mean and variance.

so bascially the idea is that we can throw out src/shogun/lib/external/SFMT and src/shogun/lib/external/dSFMT/ and use c++'s own random engines, but provide our own implementation of std::uniform_int_distribution and std::normal_distribution etc.

it's a bit unfortunate as it would have been great to be able to use standards all the way, but.... :) on the other hand i'm just saying that we only require these distributions having the very same order of the very same values because of our tests, right? conceptually the model itself suppose to work correctly if it's getting the samples of the distribution it expects (regardless of the actual values of those samples... that's why it uses a prng and not a fixed set of integers in the first place)...
it should not require to have a specific value, just that the samples are from the right distribution...

it's only ourselves, who wanna be able to test that the model is producing the very same thing with the same samples (when setting the seed)... only thing that our fixtures in the tests are for a given sample-set of a distribution on a given OS/distribution...
in other words if we could address the fact that the sample-set with a fixed seed of a random variable is different based on the compiler than we actually could use the standard all the way. i think it would be a rather big sacrifice to say that just because our tests methodology we wanna throw out half of c++11's features for random... or?
what do you think @karlnapf ?

i've just realised that say we use c++11's random features all the way in the implementation and we somehow solve the problem of testing. i'm suspecting that of course one could still argue that if we do this the problem could be that one would end up with different models using the same seed depending on the compiler OS, or? question of course is that would those model be sooooo different? on the other hand i think it's a fair presumption from the user that regardless of your compiler/system etc for the same (data, prng seed) pair you wanna end up having the same model, or?

Hi @vigsterkr , I think the root of this problem is about the use 'random' generated data in our unit test. Could we manually create some fixed mock data and use them instead of data generator like CMeanShiftDataGenerator or others?

I mean it's a feature branch anyway, so we can use result of one platform like travis-ci to verify our refactor and add fixture or some other things to make them works on different system before merge it?

@MikeLing mmmm not really... actually using CMeanShift and others would be in a way better solution than using mock data... as mock data either you have for each and every different platform/compiler... in case of random generator you can rely on the fact that they are consistent within the platform :)

btw: have you checked whether that function i pasted above generates the same set and order of elements on windows? on osx and linux it's:

4282876139
3093770124
4005303368
491263
550290313
1298508491
4290846341
630311759
1013994432
396591248
1703301249
799981516
1666063943
1484172013

@vigsterkr Hi, sorry for the late reply. Here is the output I got on window(Visual studio 2015 and window10), and you can see it has huge difference between the result from Linux and OSX

17739577640593830518
5969071622678286091
17952737041105130042
2376730765988821965
16631841681740081638
9150857099769749596
17628273237363548829
12758643431150887824
15745505721093787712
16772453002033882614
18026061444052040848
3392054037201815393
9583663648522241488
8357967283887551697
10686319701058721126

update: Oh! I got same output on Linux(ubuntu17.10, CLion based on gcc) :)

I agree viktor, not using half of c++11 because of cross platform testing is not a good idea.

I actually have an idea. What if we create a fake drop-in replacement for random (and all the distributions) that behaves deterministically. This would be purely for unit testing (not statistical testing) so it doesn't matter if the numbers are actually not "correct"
But just to ensure that random elements (forest for example) of algorithms are fixed accords platforms. Even if you fix the datasets, the algorithms might still behave different otherwise.

Then we can have statistics tests (algorithm output "makes sense") which do not rely on fixed seeds

@MikeLing ok wait this is weird. as they not only have different value but the decimal point as well... we should check on this because based on c++11 standards the random generators like std::mt19937_64 should actually be the same.

okey! it's me who has been the idiot here! the output that i've pasted was for std::mt19937 rng(100); :D i've tested with std::mt19937_64 rng(100);. i've got the same as you @MikeLing

17739577640593830518
5969071622678286091
17952737041105130042
2376730765988821965
16631841681740081638
9150857099769749596
17628273237363548829
12758643431150887824
15745505721093787712
16772453002033882614
18026061444052040848
3392054037201815393
9583663648522241488
8357967283887551697
10686319701058721126

. What if we create a fake drop-in replacement for random (and all the distributions) that behaves deterministically.

hi @karlnapf , could you tell me more about the replacement? Let's say we want to replace all the number greater than 10 to 5, and replace all the number smaller than 1 to 0?

@MikeLing are we ready with porting everywhere using std::uniform_int_distribution and std::normal_distribution? and it's only the unit tests that are failing now?

Hi @vigsterkr , sorry for the late reply.

mmm, if you mean if we can apply C++11 feature in everywhere except get the 'expected' result in the unit test, I would say not really, the NeuralNetwork test failed in a different way. And it seems like a fundamental error not some side effect.

Basically speaking the NeuraNetwork don't work at all https://pastebin.mozilla.org/9027272. And I also do some research about it, I found the error comes from this line. Normally, it should change the m_parmater like:

// Before m_params =[-0.0272588678038886943,-0.115811873679361951,0.0545971914175761991,-0.0170063438930531073,-0.00733285826272410055,0.275457538264036739,-0.140482850977850904,-0.0747966925045189662,-0.173274251285870623]

result = train_lbfgs(inputs, targets);

// After m_params=[71.3832249892633399,-922.939443864746295,247.116815510193106,564.02772070416006,282.927469091245086,650.096198731005529,-185.827272443788843,343.714997962353948,-1670.88749255393668]

But right now nothing changed after call train_lbfgs,

// Before m_params =[-0.0272588678038886943,-0.115811873679361951,0.0545971914175761991,-0.0170063438930531073,-0.00733285826272410055,0.275457538264036739,-0.140482850977850904,-0.0747966925045189662,-0.173274251285870623]

result = train_lbfgs(inputs, targets);

// After m_params= -0.0272588678038886943,-0.115811873679361951,0.0545971914175761991,-0.0170063438930531073,-0.00733285826272410055,0.275457538264036739,-0.140482850977850904,-0.0747966925045189662,-0.173274251285870623]

And furthermore, I found is because here, the rate somehow smaller than param.delta and force the loop break early then usual. Let me prove it than:

Here is the result after refactor:

) at lbfgs.cpp:482
479
480 /* The stopping criterion. */
481 if (rate < param.delta) {
-> 482 ret = LBFGS_STOP;
483 break;
484 }
485 }
(lldb) p rate
(float64_t) $19 = 0.00000000037904292365490482
(lldb) p param.delta
(float64_t) $20 = 0.00000001

And then it will jump out and return, but actually, in the same situation, it will be
(lldb) p rate
(float64_t) $18 = 0.0000014666543770853346
(lldb) p param.delta
(float64_t) $19 = 0.00000001

I don't know why we got this due to I only change the lay initiation with different random generator. We could discuss it later after I push it to the remote. Anyway, the lbfgs has too many pointers and c style things like goto. Sometimes I even don't know where the next point I should jump to, maybe we want a refactor for that?

Until now, that's only broken module after move to C++11

@MikeLing have you tried changing the seed of the prng to something else than now?
but then again this seems to be really unrelated somehow to prng itself, there's something else...

ok so now we have then everything using c++11 random, right?

based on travis apart from the neural net errors the following tests fail:

	319 - integration_meta_cpp-multiclass_classifier-random_forest (Failed)
	323 - integration_meta_cpp-clustering-gmm (Failed)```

i suppose these differences/errors are based on differences in random implementation? have you debugged this @MikeLing ?

@vigsterkr , actually if we change the generated data reference data like shogun-toolbox/shogun-data@master...MikeLing:get_rid_prng#diff-584ce98839f5d8d8329b0222352d2a77L5, all the integration test will passed on osx at least.